Since the inception of graphical user interfaces, computer painting and drawing software applications have been popular. By manipulating a mouse or another pointing device, a user can create images on the screen, which are easily edited. Typical paint programs enable a user to select pencils, brushes, spray paint cans, or other virtual tools for creating drawings. By clicking a mouse button and then dragging a pointer across the screen by moving the mouse, the user can paint or draw shapes on the screen. Of course, because the image is created on a virtual canvas supported by the computer, the user enjoys the advantages of being able to easily print and save the “painting,” undo mistakes with a keystroke, make changes in colors and fill patterns already “painted,” or simply scrap the drawing on the virtual canvas in favor of a new one, to name just a few examples.
While painting programs derive some of their popularity from these advantages of using a computer, undoubtedly some of the fascination with paint programs is the computer's response to the user movements. With a push of a finger and a move of the wrist, a user can “paint” a streak across the display that varies depending upon how the user moved a hand and/or arm in making the paint stroke. The response of the computer in such an application, therefore, presents an appealing and welcoming human-machine interface in which a user can interact with the computer in a very tangible manner.
On the other hand, while painting and drawing programs are popular, the lastingness of their appeal is somewhat limited. Although the movement of the mouse provides an appealing interaction with the computer, it is not a natural way to paint or draw. Although computers have become very commonplace in homes and schools, most people still learn to write, draw, and paint with crayons, pencils, pens, markers, brushes, and even one's fingers. The interaction of clicking a mouse button and moving a mouse across a desktop does not fully satisfy or replace the feeling one gets with using conventional implements for drawing and painting.
Moreover, not only is the human interaction with a mouse or other pointing device not the same as using a more traditional painting or drawing implement, but, with typical computer painting programs, the user never really engages the virtual paper or canvas. When a person creates a drawing or painting on real media, that person puts pen to paper, or brush to canvas. By contrast, using a typical paint program, a user moves a mouse across a horizontal desktop and watches the resulting images appear on a vertically-disposed computer screen. In other words, as is the case with most computer programs, the user engages the computer at one location, while the resulting output appears at a second location and in a different orientation.
In efforts to improve the user-friendliness of computers, some developers have created systems where users can interact more directly with a display surface. For example, the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST 10/1997:14-17,” has developed another form of “keyboardless” human-machine interface. The metaDESK includes a generally planar graphical surface that not only displays computing system text and graphic output, but also receives user input by responding to an object placed against the graphical surface. The combined object responsive and display capability of the graphical surface of the metaDESK is facilitated using infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors disposed beneath the surface of the metaDESK. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface to provide images that are visible to a user from above the graphical display surface. The IR camera can detect IR reflections from the undersurface of an object placed on the graphical surface.
Others have been developing similar keyboardless interfaces. For example, papers published by Jun Rekimoto, of the Sony Computer Science Laboratory, Inc., and associates describe a “HoloWall” and a “HoloTable” that display images on a surface and use IR light to detect objects positioned adjacent to the surface.
By detecting a specially formed object or IR-reflected light from an object disposed on a graphical display surface, the metaDESK can respond to the contemporaneous placement and movement of the object on the display surface to carryout a predefined function, such as displaying and moving a map of the MIT campus. Ultimately, however, it would be desirable to expand upon this functionality, to enable a system that would respond appropriately to ordinary objects disposed adjacent to the display surface to make the use of a personal computer even more natural. In particular, to create a more satisfying painting or drawing environment, it is desirable to create an environment where a user is able to directly engage a virtual canvas. Furthermore, to provide a user full creative expression, it is desirable to enable a user to interact with the virtual canvas using actual art tools such as a brush or tools with the shape and feel of actual art tools. In addition, it is desirable for the system to respond to the tools and generate images appearing similar to those that might result from using an actual pen, pencil, brush, or other tool on real paper or canvas.